Motor Control System Based upon Movements Inherent to Self-Propulsion

ABSTRACT

The systems and methods described herein provide hands free motor control mechanisms based on the natural and inherent movements associated with an activity of interest, and can be combined with gestures or verbal communication based upon pre-defined movements by the participant.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation in part of U.S. application Ser. No.17/144,549 filed 8 Jan. 2021, which is a continuation in part of U.S.application Ser. No. 16/560,368, filed 4 Sep. 2019, which is acontinuation in part of U.S. application Ser. No. 15/681,163, filed 18Aug. 2017, which claimed priority to U.S. provisional 62/376,878, filed18 Aug. 2016, each of which is incorporated herein by reference.

BACKGROUND INFORMATION

Developments in smaller and more powerful motors have created a varietyof motor assisted devices including skateboards, surfboards, kayaks andother human movement devices. However, these devices are controlled bythrottle mechanisms that require participant interaction and canundermine the participant experience.

Consider the act of surfing as an example. The activity of surfinginvolves the participant or surfer lying face down on a board andpaddling out past the area where the majority of waves are breaking. Thesurfer then typically waits until an appropriate wave begins toapproach. At this junction, the surfer aligns the board toward the shoreas the wave begins to crest in an effort to “catch” the wave. If thesurfer is successful in catching the wave, the surfer is pushed by thewave toward the shore and is able to perform a variety of maneuvers onthe wave. The process is typically repeated multiple times over a surfsession.

Although the above process sounds moderately easy, the process can beexceptionally tiring because paddling out through the wave is fatiguing.The process of catching the wave is also demanding, as the surfer mustget the board moving prior to the wave cresting; otherwise, the surferwill be unsuccessful in catching the wave. For the average person, thepadding difficulties and fatigue associated with the process are majorbarriers to enjoying the sport of surfing and limit the duration mostpeople can surf. Although a motor assisted surfboard has been developed,the user experience remains sub-optimal due to the need to start andstop motor assist by punching buttons on a wrist-based control device.

SUMMARY OF INVENTION

The ability to have the level of motor assistance controlled through themovements inherent to the activity, or other volitional actions, createsan enhanced user experience in activities such as surfing, kayaking, orother activities involving self-propulsion.

The systems and methods described herein provide motor controlmechanisms based on the natural and inherent movements associated withan activity of interest, and can be combined with gesture communicationbased upon defined movements by the participant. An example motorcontrol system creates an enhanced activity experience by providing theparticipant with motor assistance via a control system that does notrequire an external control device, but instead is intuitively connectedwith the activity. For example, when participating in the activity, theindividual does not have to adjust a throttle, hold a control device,push buttons, or physically interact with a control system. In practice,the system leverages intuitive and natural motions to control the motorfor an enhanced user experience. Example activities that can benefitfrom the present invention include surfing, standup paddle board,canoeing, kayaking, skate boarding, scootering, foil surfing, inlineskating and other activities that can be benefited by motor assistance.The motor control system is based upon the physical motions of theparticipant as measured by one or more of (1) kinematic sensors, whichmay include accelerometers, gyroscopic meters, and magnetometers, (2)optical systems using vision-based activity recognition, or (3) acombination of the previously mentioned systems.

In use, the system effectively identifies a signal associated with theactivity for use in motor control, while minimizing the contributionfrom noise associated with non-activity related signals or informationdue to changes in the environment. Due to the physical movement of theindividual involved in the activity, the participant's environment andbackground scene can be constantly changing and create noise artifactsthat can complicate the task of identifying the signal desired forcontrol. Example embodiments of the present invention effectively manageand minimize such artifacts and provide a quality control mechanism thatcreates the desired participant experience and is also safe to operate.Example embodiments incorporate a variety of environmental noisemitigation methods for improved performance of the system.

Because safe operation is very important, example embodiments can usethe fact that some activities require a specific sequence of events orrequire that the participant be in a defined body position. For example,in surfing, the pop-up to a standing position is preceded by a paddlingperiod. Independently or in combination, the system can use the activitystate of the participant as a necessary feature of the activation ofmotor assistance. For example, assistance with paddling can bepredicated on determination that the participant be in a paddlingposition on the board.

The use of gesture communication or other volitional actions can furtherenhance the participant experience and overall operation by enablingadditional motor assistance control during defined activities. Suchgesture communication can also be used to stop the motor in definedconditions. For example, when surfing a wave, the participant may wantadditional assistance as the wave is “petering out” but due to shoreconditions a second wave is available. In such cases, the user caninform the motor control system to add more assistance through definedgestures or other volitional actions.

As a final control mechanism, the system can also interpret and utilizevoice commands. As is the case with other control systems, management ofbackground noise or elimination of non-desired motor assistance isdesired. Thus, the system can employ a user defined wake-up word orphrase prior to accepting and executing the command. For example, thewake phrase can be defined as “Assist System.” The user can then state“Assist System, stop”.

Physical motions of the participant performing the activity can becaptured by sensors. For example, one or more motion sensors can beplaced within the paddles for activities such as kayaking, stand-uppaddle boarding or canoeing. Sensors can also be placed on theparticipant to capture the cadence of the activity. Additional sensorscan be placed on a water craft, wheeled device board to capture theinfluence of the participant's activity on the craft, or on the craft tocapture motions related to environmental noises that are unrelated tothe activity. The motion data is then processed to control the level ofassistance based upon both type of activity and the intensity of theactivity. In practice, a wearable device such as watch or similar devicecontaining an Inertial Measurement Unit (IMU) can collect the data thatis to be subjected to subsequent processing for activity state, activitycadence or rate, and combinations thereof. The resulting information canthen be converted into a motor assistance or control levels that arecommunicated to the motor.

The activity state, participant readiness as well as the cadence of theactivity can also be determined by vision-based activity recognitionmethods. Vision-based human action recognition is the process ofdetermining an activity based upon an image or series of images.Additionally, the rate of the activity or intensity can be determined bythe examination of image sequences. The vision-based activityrecognition can be done by currently available image capture systems aswell as 3D cameras.

The system and methods disclosed herein create an improved participantexperience by enabling the participant to control an assistance deviceor motor that creates an enhanced user experience in an intuitive mannerbased upon movements inherent to participation in the activity orvolitional movements.

BRIEF DESCRIPTION OF FIGURES

FIG. 1 is a schematic representation of surfer sitting.

FIG. 2 is a schematic representation of a surfer paddling.

FIG. 3 is a schematic illustration of a surfer doing a pop-up.

FIG. 4 is a schematic diagram of the test set-up.

FIG. 5 is a plot of accelerometer data from test set-up.

FIG. 6 is a plot of the gyroscopic data from the test set-up.

FIG. 7 is a plot of the gyroscopic data from the test set-up withoffset.

FIG. 8 is a plot of gyroscopic data representative of a slow paddlingcadence.

FIG. 9 is a plot of gyroscopic data representative of a medium paddlingcadence.

FIG. 10 is a plot of gyroscopic data representative of a fast paddlingcadence.

FIG. 11 is a plot of IMU data obtained during an example pop-up (firstinstance).

FIG. 12 is a plot of IMU data obtained during an example pop-up (secondinstance).

FIG. 13 is a plot of IMU data obtained during an example pop-up (thirdinstance).

FIG. 14 is a plot of IMU data obtained from paddling during a surfsession.

FIG. 15 is a schematic representation of a surfer with left arm enteringthe water.

FIG. 16 is a schematic representation of a surfer with right armentering the water.

FIG. 17 is a schematic representation of the surfer riding the wave.

FIG. 18 is a schematic of the IMU-based system.

FIG. 19 is a schematic of the vision-based system.

FIG. 20 is a schematic illustration of a combined system.

FIG. 21 is an illustration of a process for applying motor assistance inan example embodiment.

FIG. 22 is an illustration showing the difference between a conventionalfoil board and a e-foil.

FIG. 23 is an illustration of a fin based foil activation assistancesystem

FIG. 24 is a flow diagram associated with controlling the propulsionsystem.

FIG. 25 is an illustration of a mast located foil activation assistancesystem.

DESCRIPTION OF INVENTION

Motor Control refers broadly to the control of the mechanical orelectrical systems associated with an activity of interest. Motorcontrol can include adding assistance to the activity, making theactivity easier to complete, or actively stopping the activity.

Activity Associated Movement Signals refers to signals, movements,images, or information that are related to the participation in theactivity of interest and are used for motor control. These motions arepart of and inherent to the activity of interest. For example, in motorassisted surfing such an activity associated movement signal includesthe act of paddling where the degree of assistance is proportional tothe arm cadence.

Activity State refers to the general activity of the participant as itrelates to motion differences. For example, surfing is the generalactivity but is composed of the following activity states: paddling,pop-up, standing, sitting, duck-diving and being off the board.

Participant Readiness refers to the readiness of the participant toaccept, manage, experience, or enjoy the motor control provided. A stateof Participant Readiness provides a safety measure, as it ensures thatthe participant is in the correct body position or location tosuccessfully receive motor assistance without creating a dangeroussituation for the participant.

Volitional Actions, Movements or Activities refer to those motions oractions by the participant that are intentional, premeditated,deliberate, or conscious. Examples of volitional actions includegestures, speech, motions related to self-propulsion, and changes inbody position.

Gesture Control Signal refers to signals, movements, images, orinformation that are generated with the purpose of facilitating motorcontrol. For example, in motor assisted surfing such a gesture controlsignal can be the raising of one's arm with the thumb pointing up tosignal the desire for more power to the motor.

Transportation-craft tracking system refers broadly to monitoring themovement of a watercraft, surfboard, wheeled device board, or othertransportation-craft for the purpose of determining the activity of theparticipant and the general status of the transportation craft. Forexample, the paddling of a canoe results in a rocking motion of thecanoe that is indicative of the paddling cadence of the user.

Environmental Noise refers broadly to signals, movements, images, orinformation that are not related to participation in the specificactivity of interest. These noise sources or artifacts add complexity tothe system and must be managed effectively. For example, the generalswells, waves in the ocean, or other surfers all represent examples ofEnvironmental noise.

Non-Activity Movement Noise refers to movements by the individual thatare unrelated to the activity of interest. For example, such motionswhen surfing can be associated with removing hair from the face andcleaning kelp from the surfboard leash.

Activity Sequence Logic relates broadly to use of necessary prioractivities or states to facilitate motor control. For example, thesurfer must be located on the board before motor assistance should beactivated. Arm motions associated with swimming should not triggeractivation of the motor if the surfer has not completed the necessaryactivity of getting on the board.

State Determination refers to the determination of the participant'sactivity state with additional specificity. For example, when doingstand-up paddle boarding, the activity is stand-up paddle boarding, theactivity state is paddling and the state determination is left handedpaddling. Such information can be used to add a greater degree ofassistance based upon physical characteristics of the participant.

A 3D camera refers broadly to any imaging system that captures distanceinformation in conjunction with image information. These include rangecameras, a device which produces a 2D image showing the distance topoints in a scene from a specific point and stereo cameras, a type ofcamera with two or more lenses with separate image sensors or filmframes for each lens, which allows the camera to simulate humanbinocular vision, and therefore capture three-dimensional images.

Cadence Based Self-Propulsion Activities encompasses any activity wherethe user exerts effort to initiate propulsion and the experience couldbe enhanced by motor assistance but does not include bicycling.Activities can include but are not limited to surfing, standup paddleboarding, canoeing, kayaking, skate boarding, scootering, and inlineskating.

Activity cadence is the rate of performing a repeatable activity such apaddling. The rate can be variable but is defined as a metric thatincreases with increasing cadence rate.

Hands Free Operation defines a use case where the participant is notrequired to adjust the amount of motor assistance by using a throttletype control device. Because many of the above activities require theuse of hands, using hands for paddling or using the hands for holding apaddle or a steering mechanism does not count in considering whetheroperation is hands free operation.

Example Embodiments

The example embodiments described herein create an enhanced userexperience by providing a control mechanism for assistance that is basedon movements inherent to the activity. The system does not alter orinterfere in the participant's experience but rather enhances theactivity by making it easier or more enjoyable. Unlike typical controlmechanisms that can require adjustment of a throttle or a physicalactivity unrelated to participation in the activity, the present controlsystem seamlessly captures the movements of participation and addsassistance based upon the movements. The procurement of the necessarydata occurs via non-intrusive means including, as examples, simplewatches, anklets, small IMUs located in paddles or a camera mounted tocapture images of the participant. The implementation of such a systemis complex and nuanced as the participant is moving through theenvironment and many noise artifacts are present. The invention makesuse of novel developments associated with environmental noise managementfor the implementation of a safe and effective system. These conceptsprovide for improved performance relative to prior approaches byeffectively managing various noise sources unrelated to the movementsassociated with the activity of interest.

Although there are multiple means to obtain movement information, thedisclosure will use inertial measurement units and optical system asexample embodiments. Those skilled in the art will appreciate othermechanisms to obtain movement information and will be able to readilyincorporate those other mechanisms in the systems described herein.

Use and General Processing of IMU Data

The following section describes a system for determination of aparticipant's activities for the control of an assist motor by usinginformation obtained from an inertial measurement unit, referred toherein as IMU data. The described method is generalizable to allassistance activities but will be described within the context ofsurfing. For illustration purposes, the process is articulated via aseries of discrete steps but many variations are contemplated within thepresent invention. Specifically, the sequence of the steps can bechanged as needed to facilitate effective processing.

Minimization of Environmental Noise

Environmental Noise can be reduced through a variety of methods. Theinventors have discovered, and confirmed by testing, that environmentalnoise typically has a frequency content that is different than activityassociated movement signals. Additionally, information can be combinedfrom various sensors to minimize environmental noise. Using the surfingas the example activity, environmental noise is largely due to motion ofthe ocean such as swells and waves.

These noise artifacts will have a lower frequency of change than mostactivities associated with surfing. For example, the typical swell takesseveral seconds to pass while the motion associated with paddleinitiation is more rapid. Specifically, a large swell will createsignificant movement, but the movement will have a lower frequencyresponse than most surfer-initiated paddling or pop-up motions. Thus,frequency processing of the IMU date, specifically the accelerometerdata, to reduce or ignore low frequency changes can result inenvironmental noise minimization.

The incorporation of additional sources of data can be used to cancel,minimize, or reduce environmental noise. One such strategy uses anaccelerometer or IMU in or on the surf board. The accelerometer readingsthat are common to both the board and the surfer are likely due to theocean and can be removed from the data used for determination of surferactivities. The removal of these artifacts will improve the performanceof the system by elimination of a noise source. This type of commonnoise reduction can also be applied to sensors placed on the body of thesurfer because paddling results in minimal motion of the torso relativeto the magnitude of hand motion. Additionally, data for a right and leftmounted data streams can be used to eliminate those environmental noiseartifacts that are common to both data sets. Environmental noisemanagement is an important and non-trivial element in developing aneffective motor control system, especially when using accelerometerdata.

Identification of Non-Activity Movement Noise

In addition to environmental noise management, any activity can includemovements that are not associated with the main activity and thus shouldnot result in motor control activities. It is important that thesemotions are correctly identified because unintentional changes in themotor control level can be a major detriment to the participantexperience. For example, a surfing getting the hair out of one's eyes orremoving kelp from the leash are both intentional movements but not surfmotions necessitating a change in motor control. Thus, an important dataprocessing step is to effectively discriminate unrelated motions fromsurf gestures. The process can use one or multiple threshold levels onone or more sensor readings as well as the rate of change determination.

Determination of a non-activity movement versus an activity associatedmovement can be improved by looking at the response of two sensors andlooking for repeat patterns. In surfing for example, if the sensorsobserve an activity in one arm but no motion in the other, the activityis likely a non-surf motion. In contrast, repeated motions in both armswould be highly indicative of a paddling motion. In skateboarding,motion on one leg might suggest a skating push off but the lack ofsimilar motion in the other leg can be used to distinguish walking fromskating. One of ordinary skill in the art will recognize that thesevarious methods can be used independently or in combination for theeffective determination of non-activity associated movement signals

Determination of Activity State

Determination of the activity state is based upon the use ofactivity-associated signals and results in the general classification oridentification of a given activity. Examples include paddling versussurfing versus sitting during surfing. FIG. 1 shows a typical positionof a surfer “sitting” on the board. FIG. 2 shows a typical paddlingmotion, and FIG. 3 shows a typical pop-up. In practice, these activitymovements are intentional, deterministic and repeatable but the lack ofa start point or datum makes the process more difficult. In manyactivity recognition systems, the process begins with the hands or armin a stationary location followed by the action or movement. Due tomovement in the environmental and a non-zero starting location theproblem addressed by the present invention is significantly morecomplex. Additional complexity also exists due to speed differences inactivity, differences in the exact motion taken, differences in the sizeof the participant, and instrumentation differences. By usingenvironmental noise reduction methods combined with supervisedclassification techniques, embodiments of the present invention canprovide activity identification. The methods leverage the fact that theactivities of interest are based upon repeated motions that aredeterministic in nature. This type of information can be utilized todetermine the activity type through supervised classificationtechniques. For the purposes of this disclosure, supervisedclassification techniques broadly refer to machine learning tasks ofinferring a functional relationship based upon labeled training data.Supervised learning techniques can include, but are not limited to,decision trees, K-nearest neighbors, linear regression, Bayestechniques, neural networks, logistic regression, support vectormachines and relevance vector machines.

The information obtained can be pre-processed to facilitate properactivity determination. For example, in speech recognition the speedwith which the words are spoken does not influence the meaning of thewords. In many activities, the determination of the activity isdependent on the trajectory of movement and is independent of the rateof the speed of the motion. For example, paddling can be done slow orfast, while the typical pop-up occurs rapidly. Therefore, therecognition system can effectively identify the motion regardless of themotion speed.

One method useful in the present invention for accomplishing this taskis dynamic time warping. Dynamic time warping is an algorithm formeasuring the similarity between two temporal sequences which might varyin time or speed. A well-known application of dynamic time warping isautomated speech recognition. The methodology helps the recognitionalgorithms cope with different speaking speeds. In practice, dynamictime warping calculates an optimal match between two given activities bynonlinearly “warping” the time dimension to determine a measure ofsimilarity independent of the time dimension. A variety of other methodsexist to minimize the influence of motion speed differences, but dynamictime warping is a common method.

Stage Determination

Stage determination represents a further refinement in determining theactivity associated movement. Such refinements can define right fromleft arm stand-up paddling or other sub-determinations within theactivity associated movements. Such determinations can leverageadditional information such as a magnetometer as contained in a typicalIMU. A magnetometer can be used to determine the general direction oftravel by using the earth's magnetic field. Magnetometer information canbe used to know if the surfer is paddling toward shore or away fromshore. The ability to determine general board direction is valuablebecause the motor control response can be different depending upon thedirection of travel. For example, when trying to catch a wave theresponse of the motor needs to be quite quick. In contrast, the responsewhen paddling out can be slower to create a smoother transition and surfexperience.

Amount of Assistance Determination

The rate or intensity of a participant's movements can be used todetermine an amount of motor assistance. Many self-propulsion activitiesinvolve participant motions that repeat with a cadence or a rate (e.g.,paddling, rowing), that can be effectively used to inform an appropriateamount of assistance. Additionally, kinematic parameters (displacement,velocity, and acceleration) can also be used to effectively createparameters or measures that can be used for motor control and toquantify the participant's effort or desired propulsion. For example, insurfing, kayaking and stand-up paddle boarding, the motor assistancelevel can be proportional to paddling or stroke cadence. Alternativelyor in addition, kinematic parameters related to, as examples, strokedistance, stroke length stroke depth, or acceleration within a stroke,can be used to determine or refine the motor assistance level. Asanother example, in skate-boarding, the length, duration, speed, force,acceleration or other kinematic characterization of the participant'skick can be used to determine an appropriate amount of motor assistance.

Participant Readiness

The determination of participant readiness can provide an importantsafety element in the current invention, to ensure that the participantis in a suitable position to receive motor assistance. If theparticipant is not appropriately positioned with respect to theirself-propulsion device, delivery of motor assistance can pose risk tothe participant. A variety of sensor systems placed on the participant,craft, or both, can be used to determine Participant Readiness.Participant Readiness systems can comprise any combination of opticalsensors, IMU-based systems, pressure-sensing systems, or proximitysensors that are configured to determine the position of the participantwith respect to the craft. These sensors can be the same as those usedto acquire participant motion information related to self-propulsion orcan be distinct. As examples, FIG. 16 and FIG. 18 show cases where asurfer is in a position suitable to receive motor assistance, while FIG.1 shows a situation where motor activation would not be appropriate. Inan example embodiment shown in FIG. 16 , the Participant Readinesssystem can comprise an optical face-detection system that ensures thesurfer is on the surfboard prior to enabling motor assistance. In anexample embodiment shown FIG. 18 , pressure sensors (gray circles) andan IMU sensor (black rectangle) embedded in the surf board comprise theParticipant Readiness system by determining that the participant iscentered on the board, and that the surf board is in an appropriateposition to receive motor assistance. FIG. 21 shows an exampleembodiment of a method to apply motor assistance. In this example, theapplication of motor assistance is dependent on the determination ofParticipant Readiness. The Volitional Signals acquired as part of themethod can include motions from the participant that are inherent to theactivity, physical gestures, vocal commands, or other signals associatedwith the participant's volitional actions.

Gesture Determination

The determination of gesture control signals for motor control adds anadditional level of control and safety. Gesture recognition is theprocess of categorizing an intentional movement of the hand and/or armsto express a clear action. Sign language is an example of an intentionalgesture that can be recognized. In the case of determining the type ofmotor control response desired, one can define a gesture and acorresponding motor response. The user of gesture communication canenhance overall operation by enabling additional assistance controlduring defined activities. Such gesture communication can also be usedto stop the motor in defined conditions. For example, when surfing awave, the participant might want additional assistance as the wave is“petering out” but due to shore conditions a second wave is available,and the participant might want to “power” to the next wave. In such acondition, the participant can gesture communicate with the motorcontrol system for more assistance by using motions like those uses whenwater skiing. As an analogy, in water skiing, the skier will communicatewith the board driver via gestures to go faster or slower by the wave ofan arm or the direction of a thumb. Similarly, the boat motor is cutwhen the skier makes a “cut” movement over their neck. Such simplegestures can be used to automatically perform motor control in thepresent invention.

In typical gesture recognition applications, the individual is notmoving, the environmental surrounding the individual is stationary, andthe gesture has a defined start and stop. Thus, the use application addssignificant complexity to the gesture recognition process and representsan atypical application of the technology.

Processing Nuances Associated with IMU Data

Use of Surfer-Specific Information for Activity Determination

Accelerometer data can be used for activity recognition, but systemperformance can be improved if the system is trained to compensate forparticipant-to-participant differences and environmental noise isminimized. In a typical recognition system development, the algorithmsused will be developed from data obtained from a variety ofparticipants. Such a data set can include male and female participants,participants of different skill levels, and participants of differentsizes, because accelerometer data will be in influenced by theseparticipant-to-participant differences. As a simple example, considertwo surfers paddling at 1 stroke per second. The accelerations at thewrist for the longer-armed surfer can be higher than the short-armedsurfer who is paddling with the same cadence. Thus, surfer-to-surferdifferences that create variances or differences unrelated to thesurfing actives can degrade system performance.

To demonstrate this difference in accelerometer magnitude, a simple testwas conducted. A yardstick was attached to a variable speed motor andtwo IMU devices were located on the yardstick at 34 and 24 inches fromthe rotation point. A schematic of the experimental setup is shown inFIG. 4 . The distances of the IMUs from the point of rotation simulatearm lengths consistent with the arms of smaller and larger surfers. Thesystem was started and a rate of rotation mimicking a surfer's paddlingmotion obtained. The resulting data was recorded and the magnitude ofacceleration determined by taking the vector sum of the accelerationcomponents in x, y and z. Examination of FIG. 5 shows the difference inthe accelerometer magnitudes between the IMUs at different lengths.Thus, different length arms will result in magnitude differences.Magnitude differences can be problematic based upon the recognitionsystem used. For example, it the system uses an accelerometer threshold,activity recognition errors could occur. Additionally, a directpattern-based comparison using both magnitude and frequency will havedegraded performance due to magnitude differences. Such surfer-specificdifferences can be mitigated by using the training procedures describedbelow.

The system can use participant-specific training information tonormalize or compensate for participant-to-participant differences. Thetraining of the system is the process of using participant-specificinformation to improve the performance of the surf activity recognitionmethod, as well as determine the motor assistance during paddling. Anaccelerometer-based system can be trained via three related approaches.

A first approach is a general model approach where the system is trainedto recognize motions that are common to all participants followed by aparticipant-specific normalization or compensation step. This trainingstep involves entering subject-specific information. For example,participant-specific training information can include the participant'sheight or arm span, as well as foot position on the board (e.g., goofyor regular) or right handed versus left handed. The resultingparticipant-specific information is then used to compensate fordifferences that influence the accelerometer measurements for improvedsystem performance. By way of analogy, this process is related to theset-up process with speech recognition systems. Most systems require theuser to enter the language being used. This information about the userhelps the speech recognition system perform better.

A second training method involves training the system for a givenindividual, effectively creating a participant-specific training. Theprocess can entail having the owner of the system surf one or more timesso the motion characteristics of that individual are effectivelycaptured. Such a process might be useful with those that havenon-standard surf motions. Examples of non-standard surf motions includea two-armed synchronous paddling motion, or a one-armed paddling motion.

A third method is a combination system involving the two prior methods.The system has a general recognition model installed on the system, butthe model is improved over time by using participant-specificinformation. The methods can be updated and improved over time basedupon the individual participant's characteristics. This method isanalogous to algorithm updating methods used in speech recognitionsystems on the iPhone and Dragon speech recognition systems.

Gyroscopic Data for Cadence Determination

Gyroscopic sensors measure angular velocity. The units of angularvelocity are measured in degrees per second (°/s) or revolutions persecond (RPS). Because a gyroscope measures rotational velocity, thesystem is largely insensitive to arm size. Returning to the example ofthe long and short armed surfers paddling at 1 stroke per minute, theresulting gyroscopic signal would be similar. Thus, surfer-specificcompensation issues associated with gyroscopic data are decreased due tothe fundamentals of the measurement. Additionally, the rate of paddlingcadence is directly rated to the angular velocity of the arm as measuredby the gyroscope. The use of gyroscopic data can be an important elementof the system because the data is less sensitive to environmental noisedue to the fundamental nature of the measurement.

Using the same test set-up described previously, gyroscopic data wasobtained. The magnitude of the gyroscopic data was determined and isplotted in FIG. 6 . Examination of the figure shows complete agreementbetween the measured values. FIG. 7 shows the same information as FIG. 6but an offset of 300 counts was added to facilitate bettervisualization. Thus, as demonstrated by the test set-up, the gyroscopicdata is not sensitive to arm length differences.

Sequential Logic

The system can also use sequential logic regarding the time sequence ofevents. For example, in surfing, a sitting position cannot be followedby a pop-up activity because the surfer must paddle before the pop-upcan occur. Additionally, the sequence can be used to define state orawareness of the system. For example, when paddling out from shore thesystem response can be sluggish and the data transfer rate potentiallyslower. However, when the surfer turns the board to point toward shore,moves to the paddling position and starts paddling, the system can be inhigh response mode. The system needs to sense and respond to cadencedifferences and stop motor assistance if a halt activity is initiated.The halt activity occurs when one starts to paddle into a wave butrealizes that another surfer has priority on the wave. Failure to haltresults in a drop-in and a dangerous situation. Thus, the sequence ofevents preceding an activity can be used to determine a rapid response.

Use and General Processing of Image Data

As an alternative or combined approach, visual information regarding theparticipant's activity can be used for motor control. When processingthe visual information, the general goals and objectives are the same asthe IMU data, but the use of visual data content requires somealterations. In the following sections, information and details on howto process visual information collected from several types of opticalsystems will be discussed.

For the purposes of motor control, visual activity recognition in amoving environmental creates many complexities and standard visualprocessing methods work poorly. The enclosed invention addresses thesecomplexities through a series of novel combination of processing anddata procurement methodologies.

Standard Camera Systems

The system can be implemented using a variety of vision capturetechnologies including both video and still cameras with the ability torapidly capture images. Infrared cameras can also be applicable.Additionally, the system can utilize a fisheye lens to completelycapture the environment. A fisheye lens is an ultra-wide-angle lens thatproduces visual distortion. Fisheye lenses achieve extremely wide anglesof view by forgoing producing images with straight lines of perspective(rectilinear images), opting instead for a special mapping (for example:equisolid angle), which gives images a characteristic convexnon-rectilinear appearance. Varying degrees of fisheye distortion can beused. For example, a contemporary GoPro camera has some visualdistortion.

The actions of the participant can be determined using a conventionalvideo system located so that the movements of the participant can beobserved. The resulting images or image sequences can be processed fordetermination of activity associated movement signals. Vision-basedactivity recognition is the process of labeling video informationcontaining human motion with action or activity labels. For example, anaction can be decomposed into action primitives, that are aggregated tocreate an action, which is combined to create a possibly cyclic,whole-body movement referred to as an activity. For example, “left legforward” is an action primitive, whereas “running” is an action.“Jumping hurdles” is an activity that contains starting, jumping andrunning actions.

Another method for processing visual images is optical flow. Opticalflow is the distribution of the apparent velocities of objects in animage. By estimating optical flow between video frames, you can measurethe velocities of objects in the video. The resulting descriptor basedupon optical flow vectors can be used in conjunction with multi-classsupport vector machine for activity recognition.

The application of conventional activity recognition methods to a movingenvironmental is challenging due to environmental noise. In processingvideo obtained during the act of surfing, environmental noise is asignificant issue due to lack of a non-moving reference within thevisual field. For example, (1) the horizon rocks as a function of wavesand the paddling motion, (2) the surfer moves on the board relative tothe camera during all activities, and (3) the background changes due todirection of the board, other surfers and swells/waves. Most visualprocessing tools interpret the motion of an object relative to a fixedenvironmental, like a person walking on the street. The buildings arestationary and the person moves in the environmental. In many usescenarios of the present invention, the scene is not stationary creatinga more complex processing environment.

These nuances can be minimized by utilizing different processingmethodologies to minimize or correct for environmental noise. Techniquesused include 1) horizon detection, to determine the angle of the boardin the water, 2) face or upper body detection, to determine the locationof the body centerline, 3) image masking, based on spatial or spectralfeatures, to limit analysis to the arms during paddling, and 4)comparative/differential regional analyses, to identify and removemovements common to both arms during paddling.

In testing, the use of a camera with stabilization features is ofsignificant benefit. Stabilization can be provided by multiple methods,many of which are based upon the use of gimbal mounts. These systemsenable the recording of visual information that is smooth, withoutshaking effects, and maintains a constant horizon. The stabilization ofthe camera system reduces unwanted environmental noise and facilitatesactivity recognition.

Another method is to use a camera with a limited depth of focus. Depthof focus is defined as the distance between the two extreme axial pointsbehind a lens at which an image is judged to be in focus. The use of alimited depth of focus camera specifies that only objects within adefined distance will be in focus. The result is a bokeh image where thesubject is in focus and the background is blurred. As the participant ofthe activity is the critical object and one seeks to minimizeenvironmental noise, a depth of focus specific for the participant isuseful. In practice, the participant is in focus while other objectswithin the image field are blurry. No-reference image quality measurescan be utilized to effectively determine the degree of blur usinginformation derived from the power spectrum of the Fourier transform.Other methods include the use of the Haar wavelet (HAAR), modified Haarusing singular value decomposition (SVD), and intentional blurring pixeldifference (IBD) for blur detection. These methods and related methodscan be used to effectively remove the background information that isblurry due to the use of a limited depth of field camera. These methodsare typically used to sort the quality of images from a picture sequenceused create a dimensional reconstruction of an object. Thus, the usethese tools to remove background information, as in the presentinvention, is novel.

Several methods exist for the creation of bokeh images. Currenttechnology smart phones with dual rear camera arrangements, one with ahigh-resolution camera coupled with a second typically low-resolutioncamera, can create bokeh images. The combination of the two camerasallows the system to create bokeh image. Other methods exist thatinclude multiple images and masking effects.

3D Camera Systems

Environmental noise can be reduced by using a 3D camera. For thepurposes of this description, a 3D camera is a broad term that includesany image system that captures distance information in conjunction withimage information. Examples include range cameras, a device whichproduces a 2D image showing the distance to points in a environmentalfrom a specific point and stereo cameras, a type of camera with two ormore lenses with separate image sensors or film frame for each lens,which allows the camera to simulate human binocular vision, andtherefore capture three-dimensional images. Examples of commerciallyavailable 3D cameras include the Microsoft Kinect, Orbbec Astra, IntelRealsense, Stereolabs Zeb stereo camera and others. In addition to thesecameras, light field or depth maps can be created using a camera thattakes images as different focal lengths and then post process theinformation to create a 3D image. These systems operate by differentprinciples, but are able to capture distance information in conjunctionwith image information. Although these systems are typically used fordistance determination, the information can be used for environmentalnoise reduction. The system can use image information from only adefined set of distances for determination of activity associatedmovement signals. In most activities, the camera will be mounted on thefront of the object so that the participant is located between 12 and 36inches away from the camera. Thus only image data obtained at distancesbetween 12 and 36 inches is used for processing. This method effectivelycreates an information-less background of any location in the imageplane that is greater than 36 inches away from the camera.

Although not used in situations where the environmental is moving,skeletal tracking for the creation of a skeleton stick figure can beperformed. Skeleton tracking is the process of representing theparticipant in a stick figure configuration. Such a simplerepresentation can be used to simplify calculations regarding positionand cadence.

Face Detection

In addition to the use of vision-based activity recognition, facedetection can be a valuable tool in the processing method. Although facedetection is typically used for focusing applications, the invention canuse face detection as both a safety mechanism and a control mechanism.If no face is present in the image, then the motor control will initiatean immediate stop because the participant is no longer on the device. Inthe case of surfing, it can be used to determine the position of theparticipant in a paddling position. Additionally, face detection can beused to determine when a “pop-up” to a standing position has occurred.This non-conventional use of face detection has significant value increating a safe and functional system.

Motion Capture Systems

Motion capture system are typically used for computer graphicdevelopment for video games but can be repurposed into the activitydetermination for motor control in the present invention. An examplesystem can be implemented using motion capture systems that use a camerain conjunction with markers placed on the participant. In practice, theparticipant can have wrist bands with retroreflective markers or othercharacteristics that are tracked by the camera. An extension of thistechnique can be to use optical-active techniques that use LED markers.Active or passive markers can be placed on the participant to facilitatecadence and location determination.

Attached Camera Systems

Determination of the location of an arm in space can be done via acamera and IMU system attached to the arm. Thus, unlike systemspreviously described, the camera is on the arm and observing thesurrounding environment. The process integrates three types offunctionality: (1) motion-tracking: using visual features of theenvironment, in combination with accelerometer and gyroscope data, toclosely track the device's movements in space; (2) area learning:storing environment data in a map that can be re-used later; and (3)depth perception: detecting distances, sizes, and surfaces in theenvironment. Together, these generate data about the device in sixdegrees of freedom (3 axes of orientation plus 3 axes of motion) andenable the position of the device to be known in absolute coordinatespace. Such information can be used to determine the movement activitiesof the participant and for the control of the assist system. Such aposition sensor can, for example, be part of a surfer's watch anddetermine arm position changes, the direction of the board, and the riseand tilt of the board/surfer due to a wave. Such information can be usedto ensure proper motor control and to ensure an enjoyable surfingexperience.

Combination Systems

The above IMU and image based systems can be combined based upon cost,usability and convenience needs. The use of a wrist-based IMU incombination with a camera can create a system that provides accuratedetermination of activity motion. Depending upon the activity, suchinformation can be used to determine arm rotation, leg push on askateboard, and paddle cadence. As one can appreciate, a multitude ofsystem combinations exist for effectively capturing participantactivities so appropriate motor assistance can be provided.

Motor Control

As described above, the system can determine the motions of theparticipant so motor assistance can be initiated based upon the motionsthat are natural or inherent to the activity. The system also providesfor refinements beyond a binary on-off motor control. Such an on-offcontrol can be used but can create an undesired user experience. Thus,the level of assistance as a continuous function should be defined bythe participant's natural actions. The amount of assistance can beproportional to the speed or cadence of the paddling motion. Forexample, when surfing the paddle out from shore will typically have alower cadence so the level of motor assistance can be less.Additionally, the response time of the motor control unit can be lessbecause the process is relatively constant. However, when trying tocatch a wave the level of assistance can be higher if the surfer ispaddling aggressively and the ramp to full power can be faster. Thus,the maximum assistance level can be different and the overall responsetime of the system can be less. At the point the surfer catches thewave, the activity recognition system can recognize the change inposition and the motor can be turned off, maintained, or sloweddepending upon surfer preference. At the point the surfer dismounts theboard or returns to a seated position the motor can stop.

In kayaking, the level of assistance can be defined by the cadence ofthe paddling motion. In stand-up paddle board or canoeing, the level ofassistance can be proportional to the stroke rate. In skate-boarding thelevel of assistance can be proportional to the kick speed of theparticipant. Additionally, the level of assistance can be adjusted basedupon the size of the participant, the size of the device used, or otherkinematic parameters characterizing the participant's movements. A largeparticipant will likely require more power than a smaller participant.The level of assistance can also be controlled or modified by volitionalactivities, including, as examples, the use of gesture control signalsor vocal commands. For example, a “thumbs up” signal or verbal command“faster” can be used to increase the degree of motor assistance.Embodiments of the present invention thus provide hands free controlsystems that are based upon the movements inherent to the activity ofinterest with additional gesture and voice control.

In use, the exact levels of assistance can be user-defined based uponuser preferences. For example, the level of assistance desired with along board in bigger surf might be significantly higher than the levelneeded when the wave sets are far apart and small.

System Demonstration

Use of Inertial Measurement Data for Determination of Surf Activities

To demonstrate the application of the invention, an individual wasconfigured with IMUs on both wrists. An experienced surfer went throughthe characteristic motions of surfing on a surfboard in the laboratorywhen the board was elevated on a bench so a natural paddling motioncould occur. The surfer performed the following activities: paddling atdifferent cadences and performing several pop-ups. In an effort tocreate easily visualized data, the surfer stopped paddling beforeimplementing the pop-up. FIG. 8 is a plot of the gyroscopic dataobtained for the slow paddling activity. The y-axis gyroscope channel isplotted. As a gyroscope measures rotational velocity, it is a sensingsystem well suited for measuring the rotation of the arm. FIG. 9 andFIG. 10 show the same information but at faster paddling rates.Examination of all three figures shows the ability to estimate cadencevia the use of a wrist-based gyroscope.

FIG. 11 is the IMU data obtained from the first pop-up. Two additionalpop-ups are shown in FIG. 12 and FIG. 13 . Examination of the gyroscopicdata, specifically the Y-axis and the Z-axis, show a distinctiverelationship during the pop-up maneuver. The Y-axis has a significantpositive excursion while the Z-axis has a distinctive deflection but oflower magnitude. Examination of the accelerometer data reveals a numberof deflections from baseline but the identification of a common“signature” across the three pop-ups is difficult. In summary, thetesting demonstrated the ability to determine cadence from thegyroscopic data and the ability to identify a pop-up signature from thegyroscopic data. Based upon the inventor's experience in activityrecognition, the effective utilization of all the data from the IMU willresult in a robust system for motor control.

FIG. 14 shows a segment of IMU data from a sensor on the right wristduring a surf session in the Pacific Ocean. In this segment, the surferpaddled five times with the right arm, rested briefly, then paddled forsix additional strokes. Comparing data between accelerometers (leftcolumn), gyroscopes (middle column) and magnetometers (right column), itis apparent that cadence determination is substantially easier withgyroscope and magnetometer sensors. Gyroscopes and magnetometers aremore sensitive to the relatively slow changes in angular velocity andangular position that are inherent to the paddling motion. Paddling canbe detected in the accelerometer as well (particularly the Y-component),however the accelerometer is much more sensitive to high frequency noisein the body movement and “water chop” that degrade the ability tocleanly identify each stroke and the cadence in general.

Use of Visual Data for Determination of Surf Activities

To demonstrate the invention, a GoPro video of a surf session wasobtained. Several images were captured from the video to demonstrateaspects of the invention. To facilitate representation in theapplication, the color images were processed using edge detectionalgorithms and converted into black and white images. Face detection wasperformed on the images processed and is shown by the solid black box. Asimple paddle detection system can divide the image into nine panels asshown in FIG. 15 . The two vertical lines are effectively the width ofthe surfboard when the surfer is laying down. The lower horizontal linedefines the top of the surfboard. The upper horizontal line defines anupper limit where the face should be located when paddling. As paddlingis one of the first activities, this location can be defined based uponinitial images or based upon the surfer size.

The location of a face in panel 5 is consistent with paddling, see FIG.14 . Panels 4 and 6 can be examined for the identification of a movingobject using efficient optic flow algorithms. Motion detected via opticflow can be corrected for environmental noise by subtracting orregressing out motion signals determined in other panels. Alternatively,paddling motion can be inferred based on the presence of absence of armsin panels 4 and 6. Similar to face and upper body detectors, arm andhand detectors can be trained to report the likelihood that and arm/handis present or absent in each panel for each frame. The presence of anarm in Panel 6 with the concurrent lack of an arm form in Panel 4 ishighly suggestive of a left arm paddling motion. The above example is asimplistic representation of the process but is provided as anillustrative example.

FIG. 16 is an example where the face is again present in the middle ofpanel 5, and the arm is present in panel 4. The cadence of the paddlingmotion can be determined by the time difference between detection of theright and left arms.

FIG. 17 shows the results of the same processing method but where thesurfer has successfully caught a wave and is surfing. The face detectionalgorithm now detects a smaller face due to increased distance from thecamera. Additionally, the height of the face above the board hasincreased and is above the highest horizontal line. Note, due to theactivity of surfing, the surfer might look to the right or left suchthat no face is detected immediately. Thus, the absence of a face inpanel in Panel 5 but a significant object above can be used fordetection of surfing.

Although not shown, the lack of a human object on the board isindicative of the fact that the surfer has fallen off the board.

Example Embodiments

The motor control system for the various activities can be implementedin multiple ways. For simplicity of presentation, a surfing example willbe used and two general approaches presented. In a first exampleembodiment, the IMU and the processing elements are resident in a deviceon the surfer's wrist or wrists. For example, an Apple watch with asurfer motor control app can be used because the device has an IMU,display system, and communication capabilities. Such a device cancommunicate the level of motor control to the motor.

In a second example embodiment, the IMU system simply communicates theinformation to the motor control system. The systems located on thesurfer provide information to the motor control unit and the controlunit processes the information for motor control. The above system canalso benefit from an IMU located on the board as described previously.

IMU System Example

FIG. 18 is an example illustration of the IMU-based system with two IMUunits on the surfer's wrists and a third unit on the board (blackrectangle). Pressure sensors (shown as gray circles) are embedded intothe top of the surf board to detect the presence of the participant forthe determination of Participant Readiness. The use of three IMU unitscreates three data streams of information, however a single wrist unitcan be adequate. The information can be transmitted to the board viaconventional Bluetooth technology to an antenna located in the front ofthe board. Additional robustness in communication might be desired forthe system to work effectively in the water. For example, WFSTechnologies has developed a wireless system for use in the ocean, knownas the Seatooth® technology. The resulting information is communicatedto a motor control unit (not shown) that the interfaces with the motor.

In an alternative embodiment, a Bluetooth receiver, optionally includingan IMU, can be located on the ankle of the surfer. This configurationhas several advantages as the communication between the surfer and theboard can be though the surf leash thus eliminating transmissionproblems through water. Data connection between the wrist units and theankle unit can be used as a safety stop mechanism. The motor should notbe activated if the surfer's ankle is under water. Such a conditionwould be consistent with the surfer having fallen off the board or asituation where the surfer is sitting on the board. Thus, this exampleillustrates that the lack of a Bluetooth data communication can be usedas a safety mechanism.

Craft-Mounted IMU System Example

An example embodiment using transportation-craft tracking comprises asingle IMU placed on a foil surfboard. Foil surfing is a hybrid ofsurfing and hydrofoil technology. Foil surfing replaces the traditionalfin at the bottom of a surfboard with a much longer, hydrodynamicallydesigned fin called a blade. That blade is longer than the fin on anaverage surfboard and has wings at its base. Once a critical speed isreached, the wings lift the board out of the water reducing the contactarea of the board. Once the board is out of the water the participantcan “pump” the board by rocking the board up and down in a dolphin likemanner. The pumping action uses the foil blade to propel the boardforward.

A difficulty associated with foil surfing on flat water is getting theboard moving to a speed such that the hydrofoil lifts the board out ofthe water. Typically, this is achieved by some sort of towing action bya boat, person, bike or bungee. In an example embodiment, an IMU locatedin the board detects the movements of the surfer. The foil board movesback and forth on the surface of the water as the surfer paddles or usesa paddle to create speed. The back and forth motion can be sensed by theIMU and the resulting motor control system activated. The motor canremain activated until the IMU sensed a vertical motion associated withpumping as the mechanism for propulsion. Following identification of thepumping action, the motor can decrease power and the surfer can enjoy anunassisted ride. If the surfer elects to simply continue riding, this isalso possible as they will not engage in the pumping action and themotor will remain active. Additional motor control is possible by usingother volitional sensors that create a level of motor control not linkedto activities of self-propulsion. A benefit of the embodiment is theability to “self-rescue” if the surfboard again contacts the water. Insuch a situation, the system provides the needed assistance to get thefoil active again. The surfer starts paddling, and the system can againidentify the paddling motion resulting in motor activation. In thismanner of operation, the motor assistance provided to the surfer createsenough speed to effectively engage the foil.

Camera System Example

As shown in FIG. 19 , the board is equipped with a camera or video onthe board so that visual information can be obtained. The camera can beany of a variety of different types as discussed earlier. The videoinformation is communicated to the image processing system (not shown)and subsequently to the motor control system (not shown). In the exampleshown the camera is located at the front of the board but otherlocations are possible. The camera has the ability to captureinformation associated with propulsion activities, participantreadiness, and volitional movements.

Combined System Example

For illustration purposes, FIG. 20 is an example of combined system thatincludes a camera mounted in or on the board and a wrist-based device onthe surfer. The system is based upon a motion tracking system that useboth cameras and attached systems on the individual being tracked. Thecamera used has detection sensitivities beyond the visible for improvedmotion tracking. The system combines face detection with elements ofmotion tracking. The system can determine general body position basedupon face detection methods and location of the face in the cameraframe. The surfer can wear a simple wrist band that contains infraredLEDs, 1901, (enlarged for better visualization). The band can containdifferent wavelength LEDs or different encoding frequencies (on-offrates) that provide information regarding the actual location of thesurfer's wrist. For example, the LEDs on the top of the wrist can beactivated at 20 HZ while the LEDs on the side are activated at 30 HZ.The camera-LED system is effectively a motion tracking system based uponan active optical system for cadence determination.

As one can appreciate, multiple systems that combine visual data withIMU data are possible. These systems can create the needed informationfor motor control to provide an enhanced surfing experience.

The control systems, motor control systems, and activity determinationsystems described can be implemented using any of several processingapproaches, in computing hardware and software, known to those skilledin the art. As examples, contemporary smart watches can be programmed toimplement the functions described. General purpose computing systems canbe used. Special purpose processing hardware can be used, as well asspecialty controllers used in control systems for industrial and otherapplications.

Although surfing has been used as a demonstration example, those skilledin the art will recognize that the present invention can be manifestedin a variety of forms other than the specific embodiments described andcontemplated herein. Accordingly, departures in form and detail can bemade without departing from the scope and spirit of the presentinvention as described in the appended claims.

Foil Surfing Example

Foil surfing, also known as hydrofoil surfing, is a water sport thatinvolves using a hydrofoil mounted to a surfboard to lift the board outof the water and ride on the surface of the water. The hydrofoilconsists of a long, wing-like structure that extends below the boardinto the water. As the board is propelled forward, the hydrofoilgenerates lift and causes the board to rise out of the water, reducingdrag and allowing for faster, smoother rides. Foil surfing expands thetypes and locations available for surfing.

To get started with foil surfing, riders need to have a goodunderstanding of the water and wind conditions, as well as the rightequipment. Foil surfing requires specialized boards and hydrofoils thatare designed to work together to create lift and stability. Riders alsoneed to have good balance, coordination, and control, as well as theability to read the waves and adjust their speed and direction to stayon the foil.

Foil surfing has exploded in terms of popularity and includes thefollowing types of foiling.

Flatwater foiling is a type of foil surfing that is done on calm, flatwater. It is often done on lakes, rivers, or other bodies of water wherethere are no waves. Flatwater foiling is a good way to learn the basicsof foil surfing because the conditions are more stable and predictablethan in the ocean. To create enough speed to start foiling in flatwater,the rider can either paddle the board or use a boat to tow them. Themaintenance of foiling is often done by pumping the board.

Surf foiling is a type of foil surfing that is done on waves. Itinvolves riding the hydrofoil on the face of the wave. To create enoughspeed to start foiling in surf foiling, the rider needs to paddle theboard or catch a wave using traditional surfing techniques. The surfermay use standard hand paddle techniques or stand-up paddle techniquesusing a paddle to generate speed.

Downwind foiling is a type of foil surfing that is done on open waterwith the wind at the rider's back. It involves using the wind togenerate speed and ride the hydrofoil. To create enough speed to startfoiling in downwind foiling, the rider needs to position themselvescorrectly to catch the wind and use a paddle or their hands to generateadditional speed.

Winding or wind foiling is a type of foil surfing that is done on awindsurfing board or with a hand held sail. It involves using a sail tocatch the wind and generate speed, and then using the hydrofoil to liftthe board out of the water and ride on the surface. To create enoughspeed to start foiling in winding or wind foiling, the rider needs toposition themselves correctly to catch the wind and adjust the sail togenerate the right amount of power.

Kite foiling is a type of foil surfing that is done on a kiteboard. Itinvolves using a kite to catch the wind and generate speed, and thenusing the hydrofoil to lift the board out of the water and ride on thesurface. To create enough speed to start foiling in kite foiling, therider needs to position themselves correctly to catch the wind andadjust the kite to generate the right amount of power. Kite foiling isoften done in open water, and riders can cover long distances andperform aerial tricks and maneuvers once they become comfortable on thehydrofoil.

The critical step in all types of foil surfing is foil activation. Foilactivation is the process of generating enough lift with the hydrofoilto lift the board out of the water, and the board is above the surfaceof the water. Foil activation occurs when the board reaches a speed thatenables the hydrofoil to generate enough lift such that the board andthe surfer are raised above the water. For foil surfing, there is adirect relationship between speed and lift generated by the hydrofoil.As the speed of the board increases, the hydrofoil generates more lift,which causes the board to rise out of the water and ride on the surface.The hydrofoil is designed to reduce drag and increase lift as the boardmoves through the water, which means that the hydrofoil needs to bemoving through the water at a certain speed before lift can begenerated.

Generating enough speed to start foiling can be particularly challengingin conditions with little wind or small waves. In these conditions, therider may need to generate the speed through paddling or towing behind aboat. This can require a lot of effort and can be physically demanding.

The inability to create enough speed is a major limitation for manypeople interested in foil surfing. Creating enough speed is challengingbecause the hydrofoil requires a certain amount of speed to generateenough lift to lift the board out of the water. However, the difficultycontinues as the rider needs to have good balance and coordination tomaintain their balance on the board while it is rising out of the water.

Need for Balance

When riding a foil board out of the water, balance can be difficultbecause the board is suspended above the surface and can be affected bysmall changes in weight and movement. Unlike traditional surfing, wherethe board is in constant contact with the water, foil surfing requires ahigher level of balance and coordination to stay upright and in control.Riders need to be able to adjust their weight and movements in responseto changing conditions and maintain their balance.

Foil activation is the process of generating enough lift with thehydrofoil to lift the board out of the water. Foil activation occurswhen the hydrofoil reaches a certain speed and generates enough lift tolift the board out of the water. The rider needs to shift their weightforward to balance on the hydrofoil and ride on the surface of thewater. Foil activation requires good balance and coordination, as wellas the ability to adjust to changes in the conditions. Activities notassociated with maintaining balance should be eliminated, such asturning off the propulsion system via a wrist based control unit.

When foil surfing, the arms and hands play a critical role in holdingthe paddle, sail, or kite handles. Additionally, the arms and hands actas stabilizers and help the rider maintain their balance on the board.Thus, any motion not associated with controlling a propulsion device ormaintaining balance is undesired.

The Need for Propulsion Assistance

Propulsion assistance is often desired when foil surfing, especiallywhen trying to obtain foil activation because it helps the ridergenerate enough speed to activate the hydrofoil and lift the board outof the water.

A foil activation assistance device or system is an innovativetechnology that helps surfers achieve enough speed to start foiling. Thedevice is designed to provide an additional burst of power to thesurfer, enabling them to generate enough speed and momentum to activatethe foil and lift them off the water's surface. The system isparticularly useful for beginner and intermediate surfers who maystruggle to generate the necessary speed to get up on the foil. With thehelp of a foil activation assistance device, surfers can quickly andeasily get up on the foil, allowing them to experience the exhilaratingsensation of gliding above the water's surface. This technology willrevolutionize the sport of foiling and make it more accessible to awider range of surfers.

Propulsion assistance can occur in many forms, but two electricpropeller forms are (1) foil activation assist and (2) e-foiling.

Foil Activation Propulsion Assistance

Foil activation assist is the use of a propulsion system to assist withfoil activation by increasing the speed of the board, so lift can occur.The foil activation assist system is located such that when foilsurfing, the propulsion device is no longer in the water. Thisdramatically reduces drag when foiling as the propulsion system is nolonger in the water. The propulsion system can be located in manylocations, but two possible locations are the back of the board in aconventional fin location, and near the top of the mast near the bottomof the board. Importantly, the propulsion system is turned off when thesurfer is foiling.

Electric Foiling

E-foiling, also known as electric foiling, is unlike traditional foilsurfing, which relies on the rider to generate the speed needed toactivate the hydrofoil, e-foiling uses an electric motor to provide thenecessary propulsion. The propulsion device is located at the bottom ofthe mast and remains in the water at all times. This creates significantdrag and typically requires that the propulsion device be active at alltimes.

FIG. 22 is an illustration showing the differences between aconventional foil board and an e-foil board. A typical foil board weighsaround 8 lbs. and costs less than $1,000. An e-foil board, in contrast,has a cost of $8,000 and a weight of greater than 50 lbs. The rideexperience on a e-foil is dramatically different due to the weight ofthe board.

Embodiments of the present invention provide a foil activationassistance system which is used to provide an additional burst of powerto the surfer, has a propulsion system that exits the water whenfoiling, and is not active when foiling.

Problems with Current Technology Foil Activation Assist System

Two foil activation assist systems are shown in FIG. 23 and FIG. 25 . Inboth embodiments, the propulsion system is located such that thepropulsion system will generate force when the board is resting on thewater, and foiling has not occurred. However, after foil activation, thepropulsion system will be out of the water, and the board is now abovethe water. Several foil activation systems, including mast thrusters,fin-based propulsion systems, and tail thrusters, are in commercialproduction. However, the typical control mechanism is a wrist-basedcontrol device. The controller has on-off functionality and can havepower level control.

The lack of hands-free operation creates an undesirable problem whenused. When using an existing activation assist system for foil surfing,the process requires activation of the propulsion system via the wristcontroller, and the start of forward propulsion via paddling or catchingthe wind with a kite. When an appropriate speed is reached, foilactivation can occur, and the board rises above the water. At this pointin the process, the propulsion system is no longer immersed in thewater. The loss of water resistance will cause the propeller toaccelerate and make a whirring or buzzing sound as the blades spinthrough the air rather than the denser water. The whirring noise isundesirable and annoying to other surfers. The surfer may try and turnoff the propulsion system during or after foil activation, but thismovement is difficult give the balance requirements required during foilactivation and the period when foiling has occurred.

The difficulty of operating a wrist control device arises from the factthat many foil surfing activities require using both hands. For example,using a paddle, a kite, or a sail. Additionally, the start of foilingrequires a great deal of balance, as the rider is essentially standingon a small board that is suspended above the water by a foil. Thisprocess requires intense focus to make maintain fine motor control tomake small adjustments to weight distribution and foot position. Theprocess of trying to manage a manual control switch is highlyundesirable.

Improved Foil Activation Assist System

An improved foil assist system according to the present inventionenables motor control through activity-associated movement signals andtransport-craft tracking. Activity-associated movement signals refer tosignals, movements, images, or information that are related to theparticipation in the activity of interest and are used for motorcontrol. Activities associated with paddling or the raising of both armsfor sail engagement can be used to activate the motor. Additionally, thecadence of paddling can be used to control the amount of propulsionprovided. These motions are part of and inherent to the activity ofinterest and can be used for motor control.

Turning off the motor as the propulsion system begins to exit the watercan be accomplished by a transport-craft tracking system.Transportation-craft tracking refers broadly to monitoring the movementof the watercraft for the purpose of determining the activity. Thetransport-craft tracking system uses one or more sensors to detect thefoil activation, so the motor stops concurrently with the exit of thepropulsion system from the water, or immediately after the propulsionsystem has exited the water.

Sensing methods to detect foil activation and to stop the propulsionsystem before it exits the water can include environmental sensors onthe surfboard or any location above the propulsion system. Anenvironmental sensor is a device that detects and measures changes inthe surrounding physical environment, such as a change from water toair. Sensors can include conductance sensors, conductivity sensors,capacitance sensors, optical sensors, flow sensors, pressure sensors,temperature sensors, or any method or mechanism that enables thedetermination of when a sensor has moved from a water environment to anair environment. Many types of environmental sensor can be used todetermine the transition from water to air.

Sensing methods to determine when the propulsion system has left thewater can include any system that is sensitive to changes in motorperformance as the system moves from water to air. Motor performancechanges that occur as the propulsion system, typically a propeller,moves from water-to-air include changes in torque sensors, current, androtational speed.

Sensing methods to determine the rise of the board above the water caninclude an IMU or an accelerometer.

Example Embodiment of Foil Activation Assist System in Fin

FIG. 23 and FIG. 24 will be used to explain the operation of a fin-basedfoil activation system. The board is resting on the water, themotor/propulsion system is in ready mode, see FIG. 24 . Surfer 2304starts paddling and the motion is detected by wrist device 2305. Thewrist system is able to detect the activity associated movement signalsneeded to control the motor system. The wrist system wirelesslycommunicates with motor/propulsion system 2301 and the motor isactivated at a low level. Wrist system 2305 also determines the rate ofpaddling and communicates a power level or thrust level to thepropulsion unit 2301. Concurrent with motor activation, a secondarysystem is determining if the motor should be shut off due to either alack of paddle motion or the foil 2302 has created enough lift that thepropulsion unit 2301 is exiting the water is exiting the water. Sensor2306 is located above the propulsion unit and is determining waterimmersion. If either the wrist sensor system 2305 determines no paddlingmotion, or the water sensor system 2306 determines no immersion in waterthen a signal is sent to the motor/propulsion unit to stop. As shown inFIG. 23 the surfer is above the water 2302 and foiling.

In another embodiment of control configuration, the stoppage ofpropulsion by the lack of surfer paddling may be different thandescribed as the surfer may stop paddling at the start of foilactivation, but the board is just skimming the water. In this situation,the motor can remain active until the surfer is in full foil mode. Thus,the stoppage of the motor can be delayed. Additionally, the sensor candetermine the height above the water and wait unit a threshold isobtained.

FIG. 24 is a flow chart illustration of the preceding process. Thepresent invention encompasses a multitude of control configurations forvarious surfing conditions and types.

Example Embodiment of Foil Activation Assist System on Mast

FIG. 25 depicts an embodiment that uses a mast-based propulsion system2401 and a camera system 2405. The sequence of events is similar to thatdescribed in connection with FIG. 23 though with several differences.The determination of paddle activities is done by information obtainedby the camera system 2405. The camera system is able to detect theactivity-associated movement signals needed to control the motor system.The camera system or other processing capabilities are able to determineboth the start of paddling and the cadence. This information is used tocontrol the motor/propulsion unit 2401. The water immersion sensor 2406is located on the mast above the propulsion unit.

The disclosed camera system can be especially useful in kite or sailsurfing, where the body position of the surfer is important. In the caseof sail surfing, the sail must be configured to catch the wind, and thesurfer's body position must be such that they are not simply pulled intothe water. In this wind-based foil surfing situations, there is nopaddle cadence so the motor can be activated in a predefined fashionwith stoppage of the motor occurring when the motor unit is exiting orhas exited the water.

What is claimed is:
 1. An apparatus for controlling the amount ofpropulsion assistance provided to a participant engaged in foil surfingbased upon movements of the participant and environmental sensors,comprising: (a) a volitional activity sensor system configured to sensevolitional activities of the participant; (b) a motor assistancedetermination system configured to determine an amount of motorassistance to apply, responsive to the volitional activity sensorsystem; (c) a transport-craft tracking system configured to detect foilactivation , and; (d) a motor control system configured to energize amotor responsive to the motor assistance determination system, and tostop the motor when the transport-craft tracking system indicates thatfoil activation has occurred.
 2. The apparatus of claim 1, wherein thevolitional activity sensor system comprises an optical sensor.
 3. Theapparatus of claim 1, wherein the volitional activity sensor systemcomprises an IMU-based sensor worn by the participant.
 4. The apparatusof claim 1, wherein the transport-craft tracking system comprises atleast one environmental sensor configured to detect the movement of aportion of the sensor from a water environment to an air environment. 5.The apparatus of claim 1, wherein the transport-craft tracking systemdetects changes in motor performance associated with the propulsionsystem exiting the water.
 6. A method for controlling an amount of motorassistance provided to a participant during foil surfing, comprising:(a) acquiring motion information related to the participant's motionsinherent to self-propulsion; (b) determining an amount of motorassistance to provide responsive to the motion information; (c)providing motor assistance according to the amount of motor assistancedetermined in the preceding step. (d) determining the presence of foilactivation via at least one environmental sensor, and (e) stopping themotor after foil activation has been detected.
 7. The method of claim 6,wherein acquiring motion information comprises acquiring motioninformation with an optical sensor.
 8. The method of claim 6, whereinacquiring motion information comprises acquiring motion information withwearable IMU-based sensor.
 9. The method of claim 6, further comprisingsensing volitional activities of the participant, and whereindetermining an amount of motor assistance comprises determining anamount of motor assistance responsive to the motion information and thevolitional activities.
 10. The method of claim 6, comprising sensingwhen foil activation has occurred by acquiring sensor information froman environmental sensor sensitive to the movement of a portion of thesensor from a water environment to an air environment.
 11. The method ofclaim 6, comprising sensing when foil activation has occurred byacquiring sensor information from a motor sensor sensitive to themovement of a portion of the propulsion system from a water environmentto an air environment.